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The  Honorable  Bill  Chappell,  Jr. 
Chairman,  Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 

Dear  Mr.  Chairman: 


As  you  requested,  we  assessed  the  Strategic  Defense 
Initiative  Organization's  (SDIO)  National  Test  Bed  (NTB ) 
program  to  determine  the  progress  being  made  in  acquiring 
simulation  and  testing  capabilities  for  the  Strategic  Defense 
Initiative.  Although  progress  is  being  made,  NTB  acquisition 
has  not  proceeded  as  planned.  It  took  program  officials  9 
months  longer  than  planned  to  award  a  contract  to  implement 
the  final  NTB  phase;  program  objectives  to  establish  an  early 
analysis  capability  (consisting  of  computer  hardware, 
software,  and  analysts)  were  not  fully  achieved  when 
intended;  and  construction  of  the  National  Test  Facility 
(NTF ) — the  core  of  NTB  where  simulations  and  experiments  are 

to  be  managed — has  bp**™  doi  ayoA. - -  . — - - - 
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^The  purpose  of  'N39\uLs  to  provide  a  computer-based  simulation 
capability  to  assess  strategic  defense  systems  without 
actually  building  or  deploying  such  a  system*- — trnTrls^otai^'^ 
acquisition  cost  exceeds  $900  million  as  estimated  through 
fiscal  year  1993.  SDIO  initiated  the  NTB  acquisition  in 
February  1986,  and  the  program  is  currently  in  the  final 
phase — implementation,  integration,  and  operation. 

Construction  of  the  NTF  portion  of  the  NTB  started  in  March 
1988. 


NTB  is  an  essential  element  within  the  Strategic  Defense 
Initiative  program  because  it  can  potentially  provide  the 
data  considered  necessary  for  a  full-scale  development 
program  decision  in  the  early  1990s  (generally  anticipated  in 
1992).  These  data  are  intended  to  increase  confidence  and 
support  judgments  about  a  strategic  defense  system's  military 
effectiveness,  survivability,  and  cost-effectiveness. 

Although  it  is  too  early  to  determine  the  effect  of  NTB 
program  delays  on  the  overall  Strategic  Defense  Initiative 
program,  the  progress  of  NTB  acquisition  during  the  next  few 
years  will  be  critical  in  terms  of  support  to  decision¬ 
makers  . 
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Appendix  I  discusses  these  matters  in  greater  detail. 

Appendix  II  discusses  our  objective,  scope,  and  methodology. 

We  did  not  obtain  official  agency  comments;  however,  we  did 
discuss  the  contents  of  this  report  with  SDIO  officials  and 
their  comments  were  incorporated  where  appropriate. 

We  are  sending  copies  of  this  report  to  the  Secretary  of 
Defense;  the  Director,  Office  of  Management  and  Budget;  and 
other  interested  parties.  If  you  need  additional  information 
or  have  any  questions,  please  call  me  on  275-4841. 

Sincerely  yours. 


Thomas  a.  Bre 


Brew 

Associate  Director 
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In  March  1983,  the  President  called  for  a  comprehensive  scientific 
research  effort  to  develop  a  system  that  would  render  ballistic 
missiles  impotent  and  obsolete.  This  effort,  known  as  the 
Strategic  Defense  Initiative  (SDI),  is  a  multibillion  dollar 
research  program  to  determine  the  feasibility  of  developing  and 
^deploying  a  defense  system  against  nuclear  ballistic  missiles. 
^§DIO)was  formed  by  the  Secretary  of  Defense  to  manage  the  program 
^atia  explore  candidate  technologies  for  a  strategic  defense  systems 
fSIx  technologies  are  currently  in  the  concept  demonstration/ 
validation  phase  of  the  acquisition  process.  These  technologies — 
three  sensor  systems;  two  weapon  systems;  and  a  battle 
management/command,  control,  and  communications  (BM/C3)  framework — 
were  proposed  by  SDIO  as  comprising  the  first  phase  of  a 
comprehensive  strategic  defense  system. 

^Throughout  the  concept  demonstration/validation  phase,  extensive 
testing  and  evaluation  of  the  proposed  strategic  defense  system 
will  be  required  to  support  a  decision  in  the  early  1990s 
(generally  anticipated  in  1992),  on  whether  to  proceed  into  the 
full-scale  development  phase.  The  NTB  program,  initiated  by  SDIO 
in  February  1986,  is  essential  to  testing  and  evaluating  a 
strategic  defense  system,  as  well  as  its  component  technologies. 
Acquisition  of  NTB  capabilities,  however,  has  not  proceeded  as 
planned .'^cFor  example,  it  took  program  officials  9  months  longer 
than  planned  to  award  a  contract  to  implement  the  final  NTB  phase; 
program  objectives  to  establish  an  early  analysis  capability  were 
not  fully  achieved  when  intended;  and  construction  of  the  NTF 
building,  a  part  of  NTB,  has  been  delayed. 

Figure  1.1  shows  a  diagram  of  the  planned  NTB  that  is  to 
interconnect  several  Army,  Navy,  Air  Force,  Department  of  Energy, 
and  other  national  test  and  demonstration  facilities  throughout  the 
country  into  a  single,  distributed  SDI  resource.  The  core  of  NTB 
is  NTF  which  is  to  be  located  at  Falcon  Air  Force  Station, 

Colorado.  When  completed,  this  government-owned,  contractor- 
operated  facility  is  to  be  the  repository  for  SDIO's  standard 
threat  and  environment  models,  and  the  location  where  simulations 
and  experiments  of  an  entire  strategic  defense  system  are  to  be 
managed . 
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Figure  l.t :  Oleg  ram  of  the  NTB  Network 
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NTB  IS  IMPORTANT  TO  EVALUATING 
A  STRATEGIC  DEFENSE  SYSTEM 

The  Fletcher  Panel  and  the  Eastport  Study  Group — both  composed  of 
members  from  industry,  government,  and  academia — identified  the 
need  for  computer  simulation  to  test  and  evaluate  strategic  defense 
system  concepts  and  designs.  In  1983,  the  Fletcher  Panel  issued  a 
report1  stressing  that  BM/C3  architectures,  systems  architectures, 
and  major  subsystems  must  be  proposed  in  realistic  detail  and 
tested  through  simulation  to  acquire  performance  data  on  such 
complex  systems.  The  Panel  proposed  developing  a  capability  that 
would  provide  a  range  of  options  from  simulations  of  entire 
strategic  defense  systems  to  detailed  analyses  of  individual  weapon 
and  sensor  models.  In  1985,  the  Eastport  Study  Group  recommended 
that  SDIO  construct  several  different,  but  centrally  coordinated 
simulators  for  testing  and  debugging  the  strategic  defense  system 
and  evaluating  BM/C3  strategies.  The  Group  believed  that  the 
ability  to  have  confidence  in  a  total  strategic  defense  system  is 
critical,  and  to  achieve  it,  the  computer  programs  and  algorithms 
of  the  BM/C3  and  systems  architectures  must  be  tested  in  detail 
under  a  very  large  number  of  battle  variations. 


1  Eliminating  the  Threat  Posed  by  Nuclear  Ballistic  Missiles, 
James  C.  Fletcher,  Study  Chairman,  October  1983. 
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SDIO  agreed  with  both  Panel  and  Group  assessments  of  the 
importance  and  need  for  simulation,  and  initiated  the  NTB  program 
to  acquire  the  necessary  capabilities.  According  to  SDIO,  NTB 
capabilities  are  essential  in  making  an  informed  full-scale 
development  decision  on  a  strategic  defense  system  because  the 
Anti-Ballistic  Missile  Defense  Treaty  restricts  testing  of  an 
actual  system  in  space.  SDIO  considers  NTB  to  be  its  primary  tool 
for  assessing  and  validating  the  feasibility  of  integrating  BM/C3 
capabilities  with  sensor  and  weapon  systems  and  ultimately  for 
testing  system  interoperability.  NTB  is  intended  to  sufficiently 
represent  a  strategic  defense  system  so  that  the  system’s  military 
effectiveness,  survivability,  and  cost-effectiveness  can  be 
evaluated  without  actually  building  or  deploying  such  a  system. 

PROGRAM  ORGANIZATION  AND  MANAGEMENT 

SDIO  initiated  the  NTB  acquisition  in  February  1986.  In  August 
1986,  it  created  the  NTB  Joint  Program  Office  (JPO)  at  the  Air 
Force's  Electronic  Systems  Division  to  assume  program  management 
responsibility.  In  the  fall  of  1986,  JPO  established  an  office  at 
Falcon  Air  Force  Station,  Colorado  to  monitor  construction  of  the 
NTF  building  and  to  establish  an  interim  test  facility  in  the 
existing  Consolidated  Space  and  Operations  Center  building  at  the 
Station. 

SDIO  developed  a  three-phased  NTB  acquisition  strategy  that 
included  concepts  and  requirements  definition  (phase  I ) ; 
preliminary  design  (phase  II);  and  implementation,  integration,  and 
operation  (phase  III).  Phase  I  began  in  March  1986  when  SDIO 
competitively  awarded  four  $1  million  firm  fixed-price  contracts. 

In  July  1986,  SDIO  selected  two  phase  I  contractors  to  continue 
into  phase  II.  This  phase  was  completed  in  December  1986  at  a  cost 
of  $2.7  million  per  contract.  JPO  then  began  preparing  a  request 
for  proposals  for  phase  III,  with  intentions  of  awarding  a  contract 
in  April  1987.  However,  it  was  not  until  January  1988  that  JPO 
awarded  the  contract  to  Martin  Marietta  Information  and 
Communications  Systems.  Construction  of  the  NTF  building  is  a 
separate  activity  from  the  NTB  effort,  and  in  February  1988,  JPO 
exercised  a  contract  option  with  the  Kaiser  Construction 
Corporation.  Actual  construction  started  in  March  1988. 


The  total  estimated  acquisition  cost  of  the  NTB  (including  design 
and  construction  of  the  NTF  building)  is  over  $900  million  through 
fiscal  year  1993  as  shown  in  table  1.1. 
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IMPLEMENTATION  CONTRACT  AWARD  DELAYED 

JPO  awarded  the  NTB  implementation  contract  (phase  III)  in  January 
1988 — 9  months  later  than  originally  scheduled.  The  primary  reason 
for  the  delay  was  that  JPO  needed  additional  time  to  prepare  the 
requirements  because  of  technical  concerns  raised  by  SDIO  advisory 
groups.  JPO  officials  also  attributed  part  of  the  delay  to  an 
internal  debate  over  the  type  of  contract  to  use. 

JPO  solicited  input  on  the  NTB  development  approach  from  two 
advisory  groups  (consisting  of  members  from  industry,  government, 
and  academia)  formed  by  SDIO.  In  November  1986,  these  groups — the 
Eastport  Study  Group  and  the  NTB  Ad  Hoc  Advisory  Group — expressed 
the  following  concerns  about  NTB  development: 

—  NTB  contractors  were  concentrating  on  developing  and  designing 
computer  hardware  configurations  without  placing  enough  emphasis 
on  developing  associated  software. 

—  Neither  JPO  nor  the  NTB  contractors  were  developing  standards 
and  protocols  to  provide  guidelines  for  developing  simulation 
software  modules  that  could  be  integrated. 

—  JPO  was  procuring  NTB  as  a  finished  product,  instead  of 
developing  and  building  it  in  incremental  blocks. 

The  groups  recommended  that  JPO  address  these  concerns  before 
proceeding  into  phase  III — implementation. 

The  NTB  Ad  Hoc  Advisory  Group  raised  concerns  again  in  February 
1987  after  reviewing  a  draft  copy  of  the  phase  III  technical 
requirements.  Group  members  stated  that  the  draft  request  for 


APPENDIX  I 


APPENDIX  I 


proposal  was  still  too  detailed  in  terms  of  specifying  the  type  of 
hardware  to  be  provided  and  that  it  did  not  adequately  address  more 
significant  issues  such  as  software  development  and  simulation. 

For  example,  the  draft  specified  the  number  of  communication 
networks,  workstations,  and  printers  which  would  be  located  at  the 
NTF,  but  devoted  little  more  than  a  page  to  simulation 
requirements.  The  members  recommended  reducing  the  level  of  detail 
for  the  hardware  requirements  to  allow  the  contractors  to  suggest 
alternate  solutions,  and  to  more  fully  develop  the  software  and 
simulation  requirements.  JPO  revised  the  technical  requirements  by 
eliminating  several  hardware  specifications  and  adding  sections  on 
system  simulations.  JPO  also  revised  its  acquisition  strategy  by 
placing  emphasis  on  implementing  NTB  in  increments  during  the  first 
2  years,  with  additional  capabilities  to  evolve  in  later  years. 

This  was  in  contrast  to  the  phase  II  proposals  that  emphasized  a 
completed  system  at  the  end  of  5  years. 

According  to  JPO  officials,  an  internal  debate  over  the  type  of 

contract  to  use  also  contributed  to  the  delay  in  issuing  the  i 

request  for  proposals.  The  debate  centered  on  the  Air  Force 

Electronic  Systems  Division's  preference  for  a  fixed-price  contract 

versus  SDIO's  preference  for  a  cost-type  contract.  Because  of  I 

several  unknowns  associated  with  long-term  development  of 

simulations,  a  cost-plus-award-fee  contract  was  ultimately 

selected . 

EARLY  ANALYSIS  CAPABILITY  NOT 
FULLY  ACHIEVED  WHEN  INTENDED 

JPO  recognized  that  there  would  be  some  delay  in  awarding  the 
phase  III  implementation  contract.  Because  of  its  desire  to  keep 
the  program  moving,  JPO  decided  to  provide  in-house  test  and 
evaluation  support  to  NTB  users  as  an  interim  measure  before 
selecting  the  phase  III  contractor.  According  to  JPO  officials,  a 
decision  was  made  in  January  1987  to  establish  an  early  analysis 
capability  (consisting  of  computer  hardware,  software,  and 
analysts)  by  April  1987  at  the  interim  NTF  site  in  the  Consolidated 
Space  Operations  Center  building.  By  January  1988,  when  the  phase 
III  contract  was  awarded,  JPO  had  only  partially  achieved  the 
capability  that  was  desired  by  April  1987 — 9  months  ea  lier.  As  a 
result,  analytical  assistance  to  NTB  users  was  not  fully  available 
when  intended. 

JPO  had  two  specific  early  analysis  capability  objectives.  The 
first  was  to  provide  supercomputing  support  to  potential  NTB  users, 
such  as  systems  architecture  developers  and  other  SDI  program 
developers,  so  that  they  could  run  unique  simulation  programs. 

This  was  to  be  accomplished  by  leasing  time  on  a  supercomputer  at 
the  Department  of  Energy's  Los  Alamos  National  Laboratory  in  New 
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Mexico,  and  establishing  a  high-speed,  secure  communications 
network  connecting  this  computer  to  several  SDIO  sites.  The  second 
objective  was  to  develop  simulation  standards  and  protocols,  and 
NTB  operational  concepts  and  procedures,  and  to  gain  in-house 
expertise  in  using  simulations.  This  was  to  be  accomplished  by 
hiring  12  computer  analysts. 

Necessary  computer  access 
not  obtained  when  expected 

By  April  1987,  the  early  analysis  capability  was  to  include  three 
sites--Los  Alamos,  SDIO  Headquarters,  and  the  interim  NTF — all 
interconnected  by  high-speed  (56,000  bits  per  second)  secure 
communications.  By  July  1987,  the  analysis  capability  was  to  be 
expanded  to  include  the  Air  Force's  Electronic  Systems  Division  and 
Rome  Air  Development  Center,  the  Naval  Research  Laboratory,  and  the 
Army  Strategic  Defense  Command.  However,  as  of  early  January  1988 
just  before  selecting  the  phase  III  contractor,  JPO  had  only 
established  low-speed  (2,400  bits  per  second)  secure  communications 
between  the  first  three  sites.  This  resulted  in  only  limited 
access  to  the  supercomputer.  Only  low-speed  communications  were 
achieved  primarily  because  of  difficulties  in  contracting  for 
design  and  installation  of  hardware  (computers,  communication 
systems,  cryptographic  equipment,  and  workstations)  and  the 
resulting  delays  in  requesting  required  communication  security 
approvals  from  the  National  Security  Agency. 

In  January  1987,  JPO  planned  to  design  and  install  the  early 
analysis  hardware  through  existing  Air  Force  contracts  rather  than 
awarding  a  new  contract  specifically  for  this  purpose.  This 
approach  was  intended  to  expedite  achieving  the  early  analysis 
capability.  However,  Air  Force  contracting  specialists 
subsequently  concluded  that  the  work  was  outside  the  scope  of  the 
established  Air  Force  contracts.  It  was  not  until  January  1988 — 
one  year  later — that  the  design  and  installation  work  actually 
began.  JPO  had  not  done  sufficient  work  to  design  and  install  the 
interim  test  facility  equipment  so  that  timely  approval  could  be 
provided  by  the  National  Security  Agency.  As  of  March  1988,  JPO 
had  achieved  an  early  analysis  capability  equivalent  to  that  which 
was  originally  intended  in  April  1987 — 11  months  earlier--but  had 
not  yet  achieved  the  capability  that  was  intended  by  July  1987. 

Hiring  goals  for  in-house 
analysts  not  met 

JPO  also  planned  to  establish  an  in-house  analytical  capability  by 
hiring  a  cadre  of  computer  analysts.  JPO  wanted  the  analysts  to 
(1)  assist  NTB  users,  such  as  SDIO's  systems  architecture  group,  to 
analyze  alternative  strategic  defense  system  configurations,  (2) 
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develop  standards  for  integrating  software  into  the  NTB  simulation 
framework,  (3)  develop  operational  procedures  and  guidelines  for 
future  NTB  users,  and  (4)  enable  JPO  to  more  effectively  interact 
with  the  phase  III  contractor  who  would  be  charged  with  NTB 
implementation  and  operations. 

JPO  planned  to  hire  4  analysts  by  May  1987  and  a  total  of  12  by 
August  1987.  However,  as  of  March  1988,  only  two  analysts  had 
been  hired.  JPO  officials  attributed  this  hiring  delay  to 
cumbersome  personnel  procedures.  They  stated  that  because  all 
three  military  services--Air  Force,  Army,  and  Navy--contribute 
staff  to  JPO,  the  procedures  for  hiring  personnel  are  complicated. 
The  position  descriptions,  ranks,  and  composition  of  all  staff 
(including  the  analysts)  required  the  approval  of  each  service,  and 
hiring  analysts  required  more  time  than  anticipated.  Although  the 
position  descriptions  for  the  analysts  were  completed  in  January 
1987,  actual  hiring  approval  was  not  obtained  by  each  of  the 
services  until  October  1987.  By  March  1988,  actions  were  still 
underway  to  hire  nine  more  analysts,  but  JPO  officials  were  unsure 
when  the  process  would  be  completed. 

NTF  CONSTRUCTION  HAS  BEEN  DELAYED 

Construction  of  the  building  to  house  the  core  of  NTB — the  NTF — at 
Falcon  Air  Force  Station,  Colorado  has  been  delayed  because  of 
funding  constraints  and  winter  weather  conditions.  SDIO  intended 
to  start  construction  in  November  1987  (assuming  that  fiscal  year 
1988  military  construction  funds  were  appropriated  by  October  1987) 
and  complete  the  building  by  September  1989 — a  22-month 
construction  schedule.  It  actually  started  construction  in  March 
1988  (4  months  later  than  planned),  and  current  plans  are  to 
complete  the  building  in  November  1989  (2  months  later  than 
planned)  for  an  overall  20-month  construction  schedule.  SDIO 
officials  stated  that  the  start  of  construction  was  delayed  because 
it  did  not  receive  all  of  the  $100  million  in  military  construction 
funds  requested  from  the  Congress  in  fiscal  year  1988,  and  the 
funds  it  did  receive  were  not  available  when  anticipated.  In 
addition,  after  receiving  fiscal  year  1988  appropriations,  SDIO  was 
unable  to  start  construction  immediately  because  of  weather 
conditions  at  the  construction  site. 

The  conference  report  on  the  Military  Construction  Appropriations 
Act,  1988  (enacted  on  December  22,  1987)  states  that  a  phased 
funding  approach  would  be  provided  by  appropriating  $35  million. 

It  suggested  that  the  remaining  funds  be  requested  in  fiscal  year 
1989.  The  Senate  Committee  report  on  1988  military  construction 
appropriations  expressed  concern  that  the  goals  of  the  NTF  and  the 
entire  NTB  program  were  vague  and  not  fully  defined.  As  a  result, 
the  appropriations  conferees  agreed  with  Senate  bill  language  that 
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restricted  the  use  of  funds  until  certain  development  conditions 
were  met.  The  act  required  that  none  of  the  funds  were  to  be 
obligated  or  expended  until  SDIO  had  begun  certain  architecture 
work,  and  provided  reports  to  the  House  and  Senate  Committees  on 
Appropriations  on  (1)  this  architecture  work  and  (2)  the  capability 
of  the  NTF  and  other  NTB  components. 

SDIO  may  be  somewhat  optimistic  regarding  the  current  building 
construction  schedule.  The  SDIO  Director  reported  to  the  House  and 
Senate  Committees  on  Appropriations  in  January  1988  that  "worst- 
case  planning"  could  result  in  a  5-month  delay  from  the  original 
schedule.  This  would  result  in  completing  the  building  in 
February  1990.  SDIO  has  emphasized  that  to  avoid  a  break  in 
construction  and  to  maintain  the  current  construction  schedule,  it 
must  receive  fiscal  year  1989  military  construction  funds  in 
October  1988. 

Meanwhile,  JPO  plans  to  continue  using  space  in  the  Consolidated 
Space  Operations  Center  as  an  interim  test  facility.  However,  the 
Center  was  built  to  house  operations  and  control  functions  for 
several  Air  Force  satellite  programs.  Because  of  these  prior 
commitments,  JPO  is  required  to  vacate  part  of  its  space  by  January 
1989  and  the  remainder  in  January  1990.  JPO  also  anticipates  that 
growth  in  NTB  activity  will  require  additional  space  by  August 
1988.  It  therefore  plans  to  acquire  temporary  modular  buildings  to 
accommodate  this  need  at  Falcon  Air  Force  Station.  These  buildings 
are  to  be  used  for  administrative  purposes  only. 

SUMMARY 


NTB  is  an  essential  element  of  the  SDI  program  because  it  is 
expected  to  provide  the  data  considered  necessary  to  make  a  full- 
scale  development  program  decision  in  the  early  1990s.  During  the 
existing  concept  demonstration/ validation  phase,  these  data  are 
intended  to  increase  confidence  and  support  judgments  about  a 
strategic  defense  system's  military  effectiveness,  survivability, 
and  cost-effectiveness.  However,  acquisition  of  NTB  test  and 
evaluation  capabilities  has  not  proceeded  as  planned.  It  took 
program  officials  9  months  longer  than  planned  to  award  a  contract 
to  implement  the  final  NTB  phase;  program  objectives  to  establish 
an  early  analysis  capability  were  not  fully  achieved  when  intended; 
and  NTF  construction  has  been  delayed.  It  is  too  early  to 
determine  the  effect  of  these  delays  on  the  overall  SDI  program. 
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OBJECTIVE,  SCOPE,  AND  METHODOLOGY 

As  requested  by  the  Chairman,  Subcommittee  on  Defense,  House 
Committee  on  Appropriations,  we  assessed  SDIO's  NTB  program  to 
determine  progress  made  in  acquiring  the  desired  simulation  and 
testing  capabilities  for  SDI. 

We  interviewed  officials  at  SDIO  Headquarters,  Washington,  D.C.; 
the  NTB  Joint  Program  Office  at  Hanscom  Air  Force  Base, 
Massachusetts;  and  the  interim  NTF  at  Falcon  Air  Force  Station, 
Colorado.  We  reviewed  planning  and  contractual  documents,  cost  and 
schedule  information,  system  requirements  and  design  data,  and 
correspondence  concerning  the  management  and  direction  of  the  NTB 
program.  Our  review  was  performed  from  January  1987  to  March  1988 
in  accordance  with  generally  accepted  government  auditing 
standards. 
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